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Abstract. We introduce a new cryptographic tool: multiset hash functions. Unhke standard 
hash functions which take strings as input, multiset hash functions operate on multisets (or 
sets). They map multisets of arbitrary finite size to strings (hashes) of fixed length. They are 
incremental in that, when new members are added to the multiset, the hash can be updated 
in time proportional to the change. The functions may be multiset-collision resistant in that 
it is difficult to find two multisets which produce the same hash, or just set-collision resistant 
in that it is difficult to find a set and a multiset which produce the same hash. 

In particular, we introduce four multiset hash functions, each with its own advantages. 
MSet-XDR-Hash uses the XOR operation and is very efficient; however, it uses a secret key 
and is only set-collision resistant. MSet-Add-Hash uses addition modulo a large integer and, 
thus, is slightly less efficient than MSet-XOR-Hash; MSet-Add-Hash also uses a secret key but it 
is multiset-collision resistant. MSet-Mu-Hash uses finite field arithmetic and is not as efficient 
as the other two hash functions; however, MSet-Mu-Hash is multiset-collision resistant, and 
unlike the other two hash functions, does not require a secret key. MSet-VAdd-Hash is more 
efficient than MSet-Mu-Hash; it is also multiset-collision resistant, and does not use a secret 
key, but the hashes it produces are significantly longer than the hashes of the other functions. 

The proven security of MSet-XOR-Hash and MSet-Add-Hash is quantitative. We reduce the 
hardness of finding collisions to the hardness of breaking the underlying pseudorandom 
functions. The proven security of MSet-Mu-Hash is in the random oracle model and is based 
on the hardness of the discrete logarithm problem. The proven security of MSet-VAdd-Hash 
is also in the random oracle model and is based on the hardness of the worst-case shortest 
vector problem. 

We demonstrate how set-collision resistant multiset hash functions make an existing offline 
memory integrity checker secure against active adversaries. We improve on this checker such 
that it can use smaller time stamps without increasing the frequency of checks. The improved 
checker uses multiset-collision resistant hash functions. 

Keywords: incremental cryptography, multiset hash functions, set-collision resistance, multiset- 
collision resistance, memory integrity checking 



1 Introduction 

Standard hash functions, such as SHA-1 [NIS95] and MD5 [Riv92], map strings of arbitrary 
finite length to strings (hashes) of a fixed length. They are collision-resistant in that it 
is difficult to find different input strings which produce the same hash. Incremental hash 
functions, described in [BGG94], have the additional property that, given changes to the 
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input string, the computation to update the hashes is proportional to the amount of change 
in the input string. For a small change, incremental hashes can be quickly updated, and 
do not need to be recalculated over the entire new input. 

Multiset hash functions are a novel cryptographic tool, for which the ordering of the 
inputs is not important. They map multisets of arbitrary finite size to hashes of fixed 
length. They are incremental in that, when new members are added to the multiset, 
the hash can be quickly updated. Because multiset hash functions work on multisets, we 
introduce definitions for multiset-collision resistance and set-collision resistance. 

We demonstrate how multiset hash functions enable secure offline integrity checkers 
for untrusted memory. Offline memory integrity checking was introduced by Blum et al. 
[BEG'''91]. However, their implementation of offline checkers uses e-biased hash functions 
[NN90]; these hash functions can be used to detect random errors, but are not crypto- 
graphically secure. We prove that multiset hash functions are cryptographically secure, and 
show how they can be used in place of e-biased hash functions to build an offline checker 
secure against active adversaries. Furthermore, we introduce an improved checker that can 
perform better because it uses smaller time stamps without increasing the frequency of 
checks. Checking the integrity of memory is important in building secure processors which 
can facilitate software licensing and Digital Rights Management (DRM) [SCG'^OS]. 

The paper is organized as follows. Section 2 describes related work and summarizes 
our contributions. Multiset hash functions are defined in Section 3. MSet-XOR-Hash and 
MSet-Add-Hash are described in Section 4, and MSet-Mu-Hash and MSet-VAdd-Hash are 
described in Section 5. Our application of multiset hash functions to checking the integrity 
of memory is detailed in Section 6. Section 7 concludes the paper. Appendices A, B, C, 
and D prove the security of our multiset hash functions. Appendix E proves the security 
of our new memory integrity checker. 

2 Related Work and Our Contributions 

The main contribution of our work is the introduction of multiset hash functions together 
with the definition of multiset and set collision resistance. The second contribution is the 
development of a general theory leading to Theorem 1 from which we derive set-collision 
resistance for MSet-XOR-Hash, a multiset hash based on the XOR operation (addition 
modulo 2), and multiset-collision resistance for MSet-Add-Hash, a multiset hash based on 
addition modulo a large integer. The theory generalizes the results in [BGR95], where 
an XOR-based scheme is used for message authentication. Our theory holds for addition 
modulo any integer. 

Both MSet-XOR-Hash and MSet-Add-Hash use a secret key. The third contribution is 
Theorem 2 that proves multiset-collision resistance for MSet-Mu-Hash, a multiset hash 
function based on multiplication in a finite field; MSet-Mu-Hash does not use a secret key. 
The proof's basic line of thought is from [BM97] which develops message hashing based 
on multiplication in a finite field. The fourth contribution, leading to MSet-VAdd-Hash, 
is Theorem 3 proving that we may replace multiplication in the finite field by vector 
addition modulo a large integer. In [BM97] , a similar theorem is used for message hashing. 
Our theorem (and their theorem) follows directly from application of Ajtai's theorem 
[GGH96,Ajt96]. 



Our final significant contribution is that we introduce an offline checker that is cryp- 
tographically secure and which improves on the performance of the offline checker in 

[BEG+91]. 

3 Multiset Hash Functions 

This section describes multiset hash functions. We first introduce multisets. We refer to a 
multiset as a finite unordered group of elements where an element can occur as a member 
more than once. All sets are multisets, but a multiset is not a set if an element appears 
more than once. Let M be a multiset of elements of a countable set B. The number of 
times 6 G i? is in the multiset M is denoted by Mf, and is called the multiplicity of b in 
M. The sum of all the multiplicities of M is called the cardinality of M. Multiset union 
combines two multisets into a multiset in which elements appear with a multiplicity that 
is the sum of their multiplicities in the initial multisets. We denote multiset union by U 
and assume that the context in which U is used makes clear to the reader whether we 
mean set union or multiset union. 

Definition 1. Let {'H,+-}i,=n) ^^ o, triple of probabilistic polynomial time (ppt) algo- 
rithms. That triple is a multiset hash function if it satisfies: 

compression: 7i maps multisets of B into elements of a set with cardinality ~ 2™, where 

m is some integer. Compression guarantees that we can store hashes in a small bounded 

amount of memory. 
comparability: Since TC can be a probabilistic algorithm, a multiset need not always hash 

to the same value. Therefore we need =7^ to compare hashes. The following relation 

must hold for comparison to be possible: 

H{M) =n n{M) 

for all multisets M of B. 
incrementality: We would like to be able to efficiently compute 7i{M U M') knowing 
7i{M) and 7i{M'). The +7^ operator makes that possible: 

n{M u M') =n n{M) +n n{M') 

for all multisets M and M' of B. In particular, knowing only 7i{M) and an element 
b£ B, we can easily compute n{M U {b}) = n{M) +7^ H({6}). 

As it is, this definition is not very useful, because 7i could be any constant function. 
We need to add some kind of collision resistance to have a useful hash function. A collision 
for M' is a multiset M ^ M' such that n{M) =n n{M'). 

Definition 2. A multiset hash function is (multi) set-collision resistant if it is compu- 
tationally infeasible to find a (multi)set S of B and a multiset M of B such that the 
cardinalities of S and M are of polynomial size in m, S ^ M, and 7i{S) =7^ 7i{M). 



Remark. (Not needed for understanding the remainder of this paper.) To make this notion 
formal, we need to consider a family J^ of multiset hash functions (TCk , +'Hk ' — Wk ) indexed 
by a key (seed) K G IC. For TCk in ^, we denote by mx the logarithm of the cardinality 
of the set into which TCk maps multisets of B, that is rriK is the number of output bits of 
TCk- We define /Cm as the set of keys K G /C for which tjik > rn. By A{TCk) we denote a 
probabilistic polynomial time (in rriK) algorithm with oracle access to {TCk^ ~^'Hk^ —Hk)- 
The family J^ satisfies (multi)set-collision resistance if for all ppt algorithms A{.), any 
number c, and m large enough (with respect to c)/ 

(K^ICm,{S,M)^A{nK): ] 

Prob < S" is a (multi)set and M is a multiset of i? > < m^'^. 
[ such that S" / M and TCk{S) ='Hk 'Hk{M) J 

Note that because A(TCk) is polynomial in rriK, we will consider that it can only output 
polynomial sized S and M. We are disallowing compact representations for multisets that 
would allow A{.) to express larger multisets (such compact representations do not lead to 
a feasible attack in our offline memory integrity application). 

4 Additive Multiset Hash 

In this section we give an example of a construction of (multi)set-collision resistant multiset 
hash functions. Let B = {0, 1}™ represent the set of bit vectors of length m and let M be 
a multiset of elements of B. Recall that the number of times b £ B is in the multiset M 
is denoted by Mb and is called the multiplicity of b in M. Let Hk : {0, 1}™+^ -^ TZ^ be 
randomly selected from a pseudorandom family of hash functions [GGM86] . Let 

LP:in'«2™,L<n',L<2™, 

and define 

TCk{M)= HK{0,r) + '^MbHK{l,b) mod n ; ^ M^ mod L ; 

beB beB 

where r £ B is a random nonce^. Notice that the logarithm of the cardinality niK of the 
set into which TCk maps multisets of B is equal to 

rriK = log(n') + log(L) + log(2™) « 3m. 

We say two triples [h,c,r] and [h',c',r'] are equivalent, [h;c;r] =nK [^';c';r'], if and 
only if /i — Hk{0, r) = h' — Hk{0, r') modulo n and c = d modulo L. Notice that checking 
whether TCk{M) ='Hk 'Hk{M') is efficient. We define addition of two triples [h; c; r] +-Hk 
[h'] c'; r'] by the result of the computation 

[i!fx(0,r") + /i — //i^(0, r) + /i' — //i^(0, r') mod n ; c + c mod L ; r"]\^ii^B- 

Clearly, TCk{M U M') =w^ TCk{M) +'Hk TCk{M'), hence, {TCk, +nK^=HK) is a multiset 
hash. The proof of the next theorem is in Appendix A. 



^ The probability is taken over a random selection of K in fCm (denoted hy K ^- K,m) and over the 

randomness used in the ppt algorithm A{Ti.K) (denoted by {S,M) ^- A{Ti.K))- 
^ Note, the set from which r is taken could be smaller than B. 



Theorem 1. It is computationally infeasible to find a multiset M with multiplicities < n 
and a multiset M' such that the cardinalities of M and M' are polynomial sized in m, 
M / M', and nxiM) =7^^ nxiM'). 

As an example we consider n = 2 and I = m. Then the condition that a muhiset M 
has muhiphcities < 2 simply means that M is a set. This leads to set-collision resistance. 
Furthermore notice that addition modulo 2 defines xor ©. 

Corollary 1. (MSet-XOR-Hash) The multiset hash corresponding to 



■HKiM) 



Hk{0, r) © MbHKil, b); J] M, mod 2™ ; 



beB beB 



where Hx '■ {0, 1} x B ^ 2Z^ is randomly selected from a pseudorandom family of hash 
functions, is set- collision resistant. 

Notice that TixiM) keeps track of the cardinality of M. If this were not the case then 
7iK{S) and TixiM) are equivalent for any S and M with Si, = Mb modulo n = 2 for 
b £ B. This would contradict set-collision resistance. Also notice that r ^ i? is randomly 
chosen. If r was a fixed known constant, then knowledge of n tuples [M* ; TCxiM'^)] reveals 
n vectors 

^MlHK{i,b)eZ^\ 

beB 

If n = 2m then with high probability these n vectors span the vector space ^2^. This 
means that each vector in Z^ can be constructed as a linear combination of these n 
vectors [BM97]: 

a, • ( MIHk{1, 6) ) =0(0 a^Un Hk{1, b). 

i=l \beB / beB \i=l / 

Hence, a polynomial sized collision can be constructed for any polynomial sized M. 

In Appendix B we show that for n exponentially large in m, we may remove the 
cardinality Xlbe-B ^b ^o^^ the scheme altogether. By taking / = 1 and L = n = 2™" we 
obtain the next corollary. 

Corollary 2. (MSet-Add-Hash) The multiset hash corresponding to 



HKiM) 



Hk{0, r) + Y^ MbHKil, b) mod 2™ ; 



6eB 



-B 



where Hk '■ {0, 1} x B ^ TZ^-^ is randomly selected from a pseudorandom family of hash 
functions, is multiset collision resistant. 

The main difference between the MSet-XOR-Hash and MSet-Add-Hash is binary addi- 
tion without and with carry respectively. This leads to either set collision resistance or 
multiset collision resistance. 



In Appendix B we show that it is possible to replace the random nonce r by a counter 
that gets incremented on each use of TCk- This removes the need for a random number 
generator from the scheme. Moreover, shorter values can be used for r as long as the 
key is changed when r overflows; this reduces the size of the hash. Also if the weighted 
sum of the hashes Hxi^^b) in TCxiM) is never revealed to the adversary then we can 
remove Hk{0, r) from the scheme altogether. For example, in the case where the weighted 
sums are encrypted by using a pseudorandom family of permutations (see Corollary 4 in 
Appendix B). 

5 Multiplicative Multiset Hash 

A multiset-collision resistant multiplicative multiset hash can be defined as follows. Let q 
be a large prime power and consider the computations in the field GF(q). Let H : B ^ 
GF{q) be a poly-random function [GGM86], that is, no polynomial time (in the logarithm 
of q) algorithm with oracle access H can distinguish between values of H and true random 
strings, even when the algorithm is permitted to select the arguments to H (in practice 
one would use MD5 [Riv92] or SHAl [NIS95]). We define 

n{M) = llH{b)^\ (1) 

beB 

='H to be equal to =, and +-n to be multiplication in GF{q). 

Clearly, ("H, -|-^, =-^) is a multiset hash. An advantage of the scheme is that we do 
not need a secret key. Unfortunately it relies on finite field arithmetic, which makes it too 
costly for some applications. 

The proof of the following theorem is given in Appendix C, where we also define 
the discrete log (DL) assumption which says that for random y £ GF{q) and generator 
g G GF{q), it is computationally infeasible to find x such that g^ = y {x is called the 
discrete log of y) . 

Theorem 2. (MSet-Mu-Hash) Under the DL assumption, {'H,+'h,='h) '^^ defined in (1) 
is multiset collision resistant. 

Under certain assumptions we may replace multiplication in GF{q) by addition modulo 
a large number. Even though the number of output bits of the resulting multiset hash needs 
to be much larger (since it is based on 'weaker' assumptions), the overall solution becomes 
more efficient since no finite field arithmetic is needed. Let H : B ^ ZZ„, n = 2^ 
I = \/rn, be a poly-random function. Now, we define 



'rn 



n{M) = Y^ MbH{b) mod n, (2) 

beB 

='H to be equal to =, and -|-?^ to be vector addition modulo n. See Appendix D for the proof 
of the next theorem and the definition of the worst-case shortest vector (SV) problem. 

Theorem 3. (MSet-VAdd-Hash) By assuming that the SV problem is infeasible to solve 
in polynomial time, (T~C,+-}i,='H) ^-^ defined in (2) is multiset collision resistant. 
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Remark. Because H can be evaluated with oracle access to TC, Theorems 2 and 3 still hold 
for a stronger form of multiset-collision resistance, in which it is computationally infeasible 
for an adversary with oracle access to H (instead of TC) to find a collision. This is what 
allows to use a publicly available H. 

6 Integrity Checking of Random Access Memory 

We now show how our multiset hash functions can be used to build secure offline integrity 
checkers for memory. Section 6.1 explains the model, and Section 6.2 shows how our offline 
checker improves the offline checker in [BEG'''91]. 

6.1 Model 



FSM 



^ 



<^ 



load 



checker 











trusted 
state 











untrusted RAM 



read 



Fig. 1. Model 



Figure 1 illustrates the model we use. There is a checker that keeps and maintains 
some small, fixed-sized, trusted state. The untrusted RAM (main memory) is arbitrarily 
large. The finite state machine (FSM) generates loads and stores and the checker updates 
its trusted state on each FSM load or store to the untrusted RAM. The checker uses its 
trusted state to verify the integrity of the untrusted RAM. The trusted computing base 
(TCB) consists of the FSM, and the checker with its trusted state. For example, the FSM 
could be a processor. The checker would be special hardware that is added to the processor 
to detect tampering in the external memory. 

The checker checks if the untrusted RAM behaves correctly, i.e. like valid RAM. RAM 
behaves like valid RAM if the data value that the checker reads from a particular address 
is the same data value that the checker most recently wrote to that address. In our model, 
the untrusted RAM is assumed to be actively controlled by an adversary. The untrusted 
RAM may not behave like valid RAM if the RAM has malfunctioned because of errors, 
or if it has been somehow altered by the adversary. 

For this problem, a simple solution such as calculating a message authentication code 
(MAC) of the data value and address, writing the (data value, MAC) pair to the address, 
and using the MAC to check the data value on each read, does not work [LTM+00] . The 
approach does not prevent replay attacks: an adversary can replace the (data value, MAC) 
pair currently at an address with a different pair which was previously written to the 
address. The essence of an offline checker is that a "log" of the sequence of FSM operations 
is maintained in fixed-sized trusted state in the checker. 
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6.2 Improved Offline Checker 

Figure 2 sliows tlie basic put and get operations. The checker maintains two multiset 
hashes and a counter. In memory, each data value is accompanied by a time stamp. Each 
time the checker performs a put operation, it appends the current value of the counter (a 
time stamp) to the data value, and writes the (data value, time stamp) pair to memory. 
When the checker performs a get operation, it reads the pair stored at an address, and, 
if necessary, updates the counter so that it is strictly greater than the time stamp that 
was read. The multiset hashes are updated (+t^) with (a, v, t) triples corresponding to the 
pairs written or read from memory. 

Figure 3 shows how the checker implements the store-load interface. To initialize the 
RAM, the checker puts an initial value to each address. When the FSM performs a store 
operation, the checker gets the original value at the address, then puts the new value 
to the address. When the FSM performs a load operation, the checker gets the original 
value at the address and returns this value to the FSM; it then puts the same value back 
to the address. To check the integrity of the RAM at the end of the FSM's operation, 
the checker gets the value at each address, then compares WriteHash and ReadHash. 
If WriteHash is equal to ReadHash, the checker concludes that the RAM has been 
behaving correctly. Intermediate checks can also be performed with a slightly modified 
check operation [CGS+02]. First, it creates a new Timer and WriteHash. Then, as 
memory is read to perform the check, put operations are performed to update the time 
stamps in RAM and the new WriteHash. 



The checker's fixed-sized state is: 

- 2 multiset hashes: WriteHash and ReadHash. 
Initially both multiset hashes are 0. 

- 1 counter: Timer. Initially Timer is 0. 

put(a, «) writes a value v to address a in memory: 

1. Let t be the current value of Timer. Write {v,t) to a in memory. 

2. Update WriteHash: WriteHash +n= hash(a,i;,f). 



get (a) reads the value at address a in memory: 

1. Read («,i) from a in memory. 

2. Update ReadHash: ReadHash +n= hsish{a,v,t). 

3. Timer = max(TiMER, t + I). 



Fig. 2. put and get operations 



Because the checker checks that WriteHash is equal to ReadHash, substitution 
(the RAM returns a value that is never written to it) and replay (the RAM returns a 
stale value instead of the one that is most recently written) attacks on the RAM are 
prevented. The purpose of the time stamps is to prevent reordering attacks in which RAM 
returns a value that has not been written yet so that it can subsequently return stale 



initialize() initializes RAM. 

1. put(a,0) for each address a. 

store(a,i') stores v at address a. 

1. get(a). 

2. put(a,i'). 

load(a) loads the data value at address a. 

1. V = get (a). Return v to the caller. 

2. put(a,t;). 

check() checks if the RAM has behaved correctly (at the end of operation) 

1. get (a) for each address a. 

2. If WriteHash is equal to ReadHash, return true. 



Fig. 3. Offline integrity checking of random access memory 



data. Suppose we consider the put and get operations that occur on a particular address 
as occurring on a timehne. Line 3 in the get operation ensures that, for each store and 
load operation, each write has a time stamp that is strictly greater than all of the time 
stamps previously read from memory. Therefore, the first time an adversary tampers with 
a particular (data value, time stamp) pair that is read from memory, there will not be 
an entry in the WriteHash matching the adversary's entry in the ReadHash, and that 
entry will not be added to the WriteHash at a later time. 

The Timer is not solely under the control of the checker, and is a function of what is 
read from memory, which is untrusted. Therefore, the WriteHash cannot be guaranteed 
to be over a set. For example, for a sequence of store and load operations occurring on 
the same address, an adversary can decrease the time stamp that is stored in memory and 
have triples be added to the WriteHash multiple times. The ReadHash can also not 
be guaranteed to be over a set because the adversary controls the pairs that are read from 
memory. Thus, set-collision resistance is not sufficient, and we require multiset-collision 
resistant hash functions. 

The proof of the following theorem is in Appendix E. 

Theorem 4. Let W be the multiset of triples written to memory and let R be the mul- 
tiset of triples read from memory. That is, W hashes to WriteHash and R hashes to 
ReadHash. Suppose the accesses to each address are an alternation of puts and gets. 
// the RAM does not behave like valid RAM, then W ^ R. 

The following corollary shows the hardness of breaking our offline memory integrity 
checking scheme. 

Corollary 3. Tampering with the RAM without being detected is as hard as finding a 
collision W ^ R for the multiset hash function. 
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The original offline checker in [BEG^91] differs from our improved checker in two 
respects. First, the original checker is implemented with e-biased hash functions [NN90]. 
These hash functions are set-collision resistant against random errors but not against a 
malicious adversary. Secondly, the Timer is incremented on each put operation and is 
not a function of what is read from memory. The Timer is solely under the control of 
the checker. This means that the pairs that are used to update WriteHash form a set. 
Therefore set-collision resistance is sufflcient. The original offline checker can be made 
secure by using a set-collision resistant multiset hash function. 

It is worth mentioning how the two offline checkers can be optimized when there is a 
cache in the processor. Both checkers work similarly. The cache contains only data values, 
which are trusted because they are on-chip. In essence, when the cache brings in a data 
value, the checker gets the data value from the address and caches it. When the cache 
evicts a data value, the checker puts the data value to the appropriate address. The cache 
hit rate - the fraction of FSM operations that find their data in the cache - is the same 
for both schemes, and it is actually the same as for the base processor which does not 
provide memory integrity checking mechanisms. In both schemes, the principal perfor- 
mance overhead when sequences of memory operations are checked comes from reading 
and writing time stamps [008^^02]. The improved offline checker improves on the original 
checker because Timer is not incremented on every load and store operation. Thus, 
time stamps can be smaller without increasing the frequency of checks, which improves 
the performance of the checker. 

7 Conclusion 

We have introduced incremental multiset hash functions which can be efflciently updated, 
and for which the ordering of inputs is not important. Table 1 summarizes our comparison 
of the multiset hash functions introduced in this paper. In the table, we indicate whether 
the security is based on pseudorandom family of hash functions (PRF), the random oracle 
model (RO), the discrete log assumption (DL), or/and the hardness of the worst case 
shortest vector problem (SV). If hashes are to be visible to the adversary (i.e., the adversary 
can see the hashes in the trusted state, but cannot modify them), we indicate whether a 
random nonce/counter (r), or encryption is necessary. We have improved the security and 
the performance of the offline memory integrity checker in [BEG^91] as one application of 
these functions. We are currently investigating how multiset hash functions can be applied 
in other areas. 

A Proof of Collision Resistance of Additive Multiset Hash 

Let Qm be the family of matrices with 2™^^ rows, / columns, and entries in ^n (recall 
L ^ n ^ 2™). Let Hk be a random matrix in Qm = {Hi, H2, H3, . . .}. Notice that Hk is 
the i^-th matrix in Qm- We assume that this matrix, or equivalently its label K, is secret 
and only accessible by the secure processor. The family of matrices Qm from which Hk is 
selected is publicly known. 

The rows of Hk are labeled by a; G {0, 1}™+^ and denoted by Hk{x). This represents 
Hk as a function from x G {0, 1}™-+! to .^„, the set of vectors with length / and entries 
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comput. 
efficiency 
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visible hash 



MSet-XOR-Hash 
MSet-Add-Hash 
MSet-Mu-Hash 
MSet-VAdd-Hash 



set 

multiset 
multiset 
multiset 



Y 
Y 

N 
N 



PRF 
PRF 
RO/DL 
RO/SV 



+ 
+ 
+ 



original 
both 
both 
both 



r/encryption 
r/encryption 



Table 1. Comparison of the Multiset Hash Functions 



in TZn- In practice, Kk is not a completely random matrix over TZ^ but Hk is selected 
from a pseudorandom family of functions. We address this issue as soon as we are ready 
to formulate a proof of Theorem 1 . 

The following theorem is about the probability that an adversary finds a collision for 
some multiset M' . The probability is taken over random matrices Hk in Qm {Hk <— Qm) 
and the randomness of the random nonce used in IIk- 

Theorem 5. Let M and M' be multisets of B. Let d be the greatest common divisor^ of n 
and each of the differences |Mfe — M^|, b £ B. Given knowledge of u tuples [M* ; TixiM^)], 
the probability that M is a collision for M' is at most v? jT^ + [dirif . 

We first introduce some notation. Let u(r, M) be the vector of length 2™'+-'^ defined by 

v{r, M)(o,fe) = 1 if and only if 6 = r 

and 

t;(r,M)(i,;,)=Mf,. 

Let v{M) be the vector of length 2™+i defined by i'(M)(o,6) = and v{M)(xf>) = ^h- 

Lemma 1. (i) Knowing \M ; 'Hi^(M)] is equivalent to knowing 

[v{r,M) ; v{r,M)HK mod n]. 

(ii) TiKiM) =nK Ti-KiM') if and only ifv{M)HK = v{M')Hk modulo n and J2beB ^b = 
J2beB ^'b modulo L. 

Proof. Notice that v{r, M) encodes r, M, and, hence, the cardinality X^^g^ M^ of M, and 
notice that 



Hx(M) 



v{r,M)HK mod n ; > Mb mod L ; 



beB 



The lemma follows immediately from these observations. 



The greatest common divisor of with a positive integer i is equal to i. 
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Suppose that an adversary learns u tuples [M* ; TCxiM''')] or, according to Lemma 
l.(i), u vectors v{r^,M^) together with the corresponding v{r^,M^)HK modulo n. Let A 
be the u x 2™"'"^ matrix with rows f(r*,M*). Then the matrix with rows v{r^, M^)Hk 
is equal to AHk- Clearly, A modulo n has full rank over ^„ if all r* are different. The 
probability that there are two equal r*'s is at most v? jl™" . 

Lemma 2. The probability that the r'^'s corresponding to matrix A are all different is at 
least 1 - mV2™. 

By Lemma l.(ii), in order to find a collision for M', the adversary needs to find a 
multiset M ^ M' such that v{M)Hk = v{M')Hk modulo n and such that the cardinalities 
of M and M' are equal to one another modulo L. The next three lemmas show how difficult 
this is for the adversary if he is in the situation of the previous lemma. 

Lemma 3. Let M and M' be multisets of B. The probability that v{M)Hk = v{M')Hk 
modulo n is statistically independent of the knowledge of a full rank matrix A over TZ^ 
corresponding to different r* 's and the knowledge of h = AHk modulo n. 

Proof. W.l.o.g. (after reordering the columns of A and the corresponding entries of v{M) — 
v{M') and corresponding rows of Hk) matrix A has the form A = {I ^^), where I is the 
ux u identity matrix, and v{M) — v{M') has the form (0 v), where is the all zero vector 
of length u. Denote the top u rows of H^ by H^ and let H]^ be such that 



^x = ( ^f 



Clearly, the equation h = AHk modulo n is equivalent to 

h = H% + A^H]^ mod n. (3) 

The equation = {v{M) — v{M'))Hk modulo n is equivalent to 

= vH]^ mod n. (4) 

Straightforward counting tells us that Pro6{(4)|(3)} is equal to the # of matrices H]^ 
satisfying (4) divided by the total # of matrices H]^. This is in turn equal to the jj^ of 
matrices Hk satisfying (4) divided by the total ^ of matrices Hk, which is Pro6{(4)}. 

Lemma 4. Let M and M' be multisets of B. Let d be the greatest common divisor of 
n and each of the differences jM^ — M'^\, b £ B. Then {v{M) — v{M'))Hk modulo n is 
uniformly distributed in dTZ^. 

Proof. To prove this lemma, we show that each entry of {v{M) — v{M'))Hk modulo n is 
uniformly distributed in dZZn- Let y represent one of the columns of Hk and define for 

(3 G !Sn the set 

Cp = {y: {v{M) - v{M'))y = (3 mod n}. 

Since d divides each entry of v{M) — v{M'), it also divides the product (v{M) — v{M'))y, 
hence, C^ = if /3 is not divisible by d. Since d is the greatest common divisor of n and 
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each of the entries of t;(M) — v(M'), there exists a vector y such that (v{M) —v{M'))y = d 
modulo n. This proves that C/j 7^ if and only if d divides (3. For a fixed column y' ^ Cj3 ^ 0, 
the mapping y £ Cjs ^ y — y' £ Cq is a bijection. Hence, the non-empty sets Cp have equal 
cardinality. We conclude that each entry of {v{M) — v{M'))Hk modulo n is uniformly 
distributed in d!Zn. 

Lemma 5. Let M and M' he multisets of B. Let d he the greatest common divisor of n 
and each of the differences \Mb — M^\, b £ B. Given knowledge of a full rank matrix A 
over TZn corresponding to different r* 's and given knowledge of h = AHk modulo n, the 
prohahility that v{M)Hk = v{M')Hk modulo n is equal to {d/nY. 

Proof. By Lemma 3, since matrix A corresponds to different r*'s and {v{M) — v{M'))rQ ^i\ = 
0, the probability that the randomly chosen matrix Hk satisfies = {v{M) — v{M'))Hk 
modulo n is independent of the knowledge of /i = AH}^ mod n. By Lemma 4, since Hk 
is uniformly distributed, {v{M) — v{M'))Hk is uniformly distributed in dZ^. Hence, the 
probability that = {v[M) — v[M'))Hk mod n is equal to one divided by the cardinality 
of dTZ,^^ which is equal to {d/n) . 

Combining Lemmas 2 and 5 proves Theorem 5. To prove Theorem 1 we need the 
following extra lemma. 

Lemma 6. Suppose that v{M) = v{M') modulo n, 'YlbeB^b = Y2beB ^b f^odulo L, 
the cardinalities of M and M' are < L, and that the multiplicities of M are < n. Then 
M = M'. 

Proof. If the cardinalities of M and M' are equal modulo L and < L then 

Y,Mb = Y.Mi (5) 

beB b€B 

If all entries of v{M) are < n and v(M) = v(M') modulo n, then 

Ml = Mb + f3bn, b£B, (6) 

for integers (5b ^ 0- Combining (5) and (6) proves ^5^^ /?& = 0, hence, all Pb = 0. We 
conclude that M = M'. 

Now we are ready to prove Theorem 1. 

Proof. Let A(TCk) be a probabilistic polynomial time (in rrix ~ 3m) algorithm with 
oracle access to {'Hk,+Hk'='Hk)- Then A{7iK) can gain knowledge about at most a 
polynomial number u{m) tuples [M* ; T-Ck{M^)] (here u{.) denotes a polynomial). Fur- 
thermore, A(TCk) can search for a collision among at most a polynomial number t(m) of 
pairs {M,M'), where M and M' are multisets, M 7^ M' , and M has multiplicities < n. 
According to Theorem 5, the probability that A(TCk) finds a collision is at most 

t(m)(u(m)V2" + (d/n)'). 
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Since A(TCk) can only compute polynomial sized multisets, the cardinality of the 
multisets M and M' are < L ~ 2™". This allows us to apply Lemma 6 and conclude that 
7^ {v{M) — v{M')) modulo n. Hence, the greatest common divisor d oin and each of the 
differences |Mf, — M^|, 6 G i?, is at most n/2. This leads to 

(d/n)' < 2~K 

Let c > be any number and suppose that 2"' > m~'^, or equivalently, I < clogm. Notice 
that each of the differences |Mf, — M^| is polynomial sized in m, hence, d is polynomial 
sized in m and there exists a number e > such that d < mP for m large enough. This 
proves 

which is at most m"'^ for ra large enough. We conclude that the probability that A(J~iK) 
finds a collision is at most ra~'^ for m large enough. This proves Theorem 1 for random 
matrices Hk- 

Remark. The theorem also holds for a pseudorandom family of hash functions represented 
as matrices. Suppose that an adversary can compute a collision with a significant proba- 
bility of success in the case where a pseudorandom family of hash functions is used. We 
have just shown that an adversary has a negligible probability of success in the case where 
random hash functions are used. Hence, with a significant probability of success he is able 
to distinquish between the use of pseudorandom hash functions and the use of random 
hash functions. This contradicts the definition of pseudorandomness, see [BGR95] for a 
detailed proof of a similar result. 

B Variants of Additive Multiset Hash 

A few interesting variants of TIk exist. Suppose that v{M) = v{M') modulo n and that 
the multiplicities of M and M' are < n. Then clearly M = M' . Hence, we do not need 
Lemma 6 in the proof of Theorem 5. This means that the proof of Theorem 5 does not 
depend on the cardinalities of M and M' to be equal modulo L. We can remove the 
cardinality X^tg^ M5 from the scheme altogether. For example, for n exponentially large, 
the cardinalities and in particular the multiplicities of M and M' are < n. This proves 
Corollary 2. An other example is n = 2 and both M and M' are sets, which proves the 
the main result of [BGR95] . 

Secondly, it is possible to replace the random nonce r by a counter that gets incre- 
mented on each use of Tix, or by any other value that never repeats itself in polynomial 
time. This guarantees with probability 1 that the matrix A corresponds to different r*'s 
(see Lemma 2). This removes the need for a random number generator from the scheme. 
Moreover, shorter values can be used for r as long as the key is changed when r overflows; 
this reduces the size of the hash. 

If li = then the proof of Theorem 5 does not depend on matrix A and its correspond- 
ing r*'s. Similarly, if sums of hashes, 

HK{^,r) + ^Mi,HK{l,b) mod n, 
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are hidden from the adversary (he knows which multiset M is being hashed, but not the 
value of the sum of hashes) then we can remove Hk{0, r) from the scheme altogether. As 
the following corollary shows, complete hiding is not necessary. We can use a pseudorandom 
permutation to hide sums of hashes. 

Corollary 4. (Permuted-MSet-XOR-Hash) The multiset hash corresponding to 



nK,K'{M) 



\beB J beB 



mod 2" 



where Hk '■ {0, 1} x B ^ TZ"^ and Pk' are randomly selected from a pseudorandom family 
of hash functions and permutations, is set-collision resistant. 

(Permuted-MSet-Add-Hash) The multiset hash corresponding to 



'Hk,k'{M) 



PK'[Y.MbHK{l,b) mod2™j 

\b£B ) 



where Hk '■ {0, 1} xS ^ 'ZZ2-m. and Pk' are randomly selected from a pseudorandom family 
of hash functions and permutations, is multiset- collision resistant. 

Notice that the multiset hashes are incremental because Pk' is a permutation and, 
hence, invertible. 

Proof. We first consider a random function Pk'- Suppose that the adversary learns u 
tuples [M* ; Tix^K'iM'^)]- As in Lemma 2, the probability that two permuted sums of 
hashes in the u tuples are equal is at most u^/2™. If all of them are unequal to one 
another then matrix AHk (defined without the part corresponding to the random nonce) 
is uniformly distributed and not known to the adversary (since Pk' is a random function) . 
Hence, the probability that v{M)Hk = v{M')Hk modulo n is statistically independent of 
the knowledge of the adversary. This can be used instead of Lemma 5 to prove Theorems 
5 and 1. This result also holds for a pseudorandom family of permutations -Pfc, see the 
remark at the end of the proof of Theorem 1 in Appendix A. 

C Proof of Collision Resistance of Multiplicative Multiset Hash 

In the following lemma A{.) is a probabilistic polynomial time (in logg) algorithm which 
outputs weights*^ wi, . . . ,Wu G ^q-i for a polynomial number of random inputs xi, . . . ,Xu G 
GF{q) such that 1 = Y\^ x^' with probability at least p. We show that if such an algorithm 
exists then we can break the DL problem in GF{q) in polynomial time with probability 
at least p. 

Lemma 7. Let A{.) he a ppt algorithm such that there exists a number c such that for 
u < {logqf, 

Prnh / (^^ ^ GFiq))f^„ {w, G Z,^i)U - A^U • • • , X J : 1 

ii = n.<% ^^m^o, y^\w,\ < (logqr j-^- ^' 



Not all equal to zero and each of them bounded by a polynomial number. 
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Let g be a generator of GF{q) . Then there exists a probabilistic polynomial time (in log q) 
algorithm A'{.) such that 

Prob{y ^ GF{q),x^ A'{y) : y = 5"} > p/ilogqf. 

In words, given a random y £ GF{q), we are able to find the discrete log of y in GF{q) 
with probability at least p/{logqY. 

Proof Let y ^- GF(q). Select a polynomial number u of random elements ri, . . . ,r„ in 
TZq-x and j G {1, . . . , n} and compute 

Xj = yg^^ and Xj = g''^ for i / j. 

Compute (tfi, . . . ,!««) ^- A{x\^ . . . ,Xu)- Since by construction the XjS have been chosen 
uniformly at random, we know that with probability at least p the weights wi, . . . ,Wu G 
TZq^x are computed such that they are not all equal to zero, \wj\ < (logg)"^, and 

l = Y[xf^ =y'"^g^^''^'"\ (8) 

i 

Since the u inputs are in random order, the probability that Wj 7^ is at least 

l/u>{logq)-'. 

Suppose that Wj 7^ 0. Let d be the greatest common divisor between Wj and q — 1. Then^ 
Wj/d is invertible in TZq^x. By using the Chinese remainder theorem (assuming that we 
know the factorization of g — 1), we are able to compute the inverse of Wjjd in TZq^x in 
polynomial time. Denote this inverse by w'. From (8) we infer that 



V =9 



-""j E, nwi 



Notice that if y = 5^ and y = g^, then g = g^, that is dt = s modulo q — 1. Recall 
that d divides q — 1. For this reason d must also divide s. Let d' = {q— l)/(i and s' = s/d. 
Both can be computed in polynomial time as we have shown. Now y can be expressed as 
one of the roots 

where < j < d — 1. Since d < \wj\ < (logqY, each of the roots can be checked in 
polynomial time. This proves the lemma. 

The DL assumption states that for all ppt algorithms A{.), any number c, and Q large 
enough, 

^ , f Q > Q is a prime power, q generates GF(q), ^^ , ,, , _^ 

Prob< „^, . ,, ' ^':y = g}<(logq)''. 

\y^GF{q),x^A{q,y) yj-vsy; 

We are ready to prove Theorem 2. 



^ Division / denotes division over integers, not over 7Zq-\ (since d has no inverse in 7Zq-\, we can not 
divide Wj by d in 7Zq-\). 
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Proof. Suppose that there exists a number c and a probabihstic polynomial time algorithm 
B{H), which runs in time u = (logqY, with access to a random oracle H which outputs 
with probability p > 1/ua collision M for M' . That is, M 7^ M' , M and M' are polynomial 
sized < u, and 

n{M) = Yl H{b)^' = Y[ H{bf^b = n{M'). 
beB beB 

This means that 



n H(by 



beB 

there is a polynomial number M^'s and M^'s unequal to zero, for all b ^ B the absolute 
value \Mb — M^| < li is polynomial sized, and there exists ab € B such that Mi, — M^ 7^ 0. 

Let C be an algorithm that goes from GF{q)^ to B ^ GF(q), where B -^ GF{q) 
denotes the set of oracles with inputs in B and outputs in GF{q). C is chosen such that 
C(xi, . . . ,Xu) returns xi when it is called for the first time on some input yi, X2 when it 
is called for the first time on some input y2 different from yi, and so on. 

When xi, . . . ,Xu are chosen randomly, C{xi, . . . , Xu) cannot be distinguished from a 
random oracle by B because B cannot query C more than u times. Therefore, if we let A 
be the composition of B and C, ^ is able to find a collision for 7i with probability p when 
its inputs are chosen uniformly at random. Moreover, ^ is a ppt algorithm satisfying (7), 
so by Lemma 7, A can break the discrete log problem in GF{q) in polynomial time with 
probability at least p/{logqY > (log (7)"^'^. This contradicts the DL assumption. So B does 
not exist, which proves multiset-collision resistance. 

Because oracle access to H is stronger than oracle access to 7i, this proves Theorem 2 
when H \s a random oracle. The result carries over to poly- random functions because they 
are indistinguishable from random functions by ppt algorithms. 

Remark. Supposing that H \s a random oracle is a strong assumption. Compared to 
the MSet-XOR-Hash and MSet-Add-Hash we do not need a secret key (as the seed of a 
pseudorandom family of hash functions) at all. We refer to [BR93] for a discussion into 
what extent the random oracle assumption can be met in practice. 

D Proof of Collision Resistance of Vector Additive Multiset Hash 

If r is a fixed constant in the MSet-Add-Hash, then we are again vulnerable for the attack 
described for the MSet-XOR-Hash, where r is a fixed constant. The main difference is 
that the attack is not modulo n = 2 but modulo n = 2™". This means that the linear 
combination may lead to a collision with large multiplicities. This would give a non- 
polynomial sized collision and does not defeat the multiset collision resistance. It turns 
out that this problem is related to a weighted knapsack problem (see also [BM97]). In 
this sense MSet-Add-Hash remains multiset collision resistant, even if the pseudorandom 
family of hash functions Hk is replaced by a single random function avoiding the use of 
a secret key as in MSet-Mu-Hash. 
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The weighted knapsack (WK) assumption is defined as follows. For all ppt algorithms 
A{.), any number c, q large enough, and u < {logq^, 

[0 = 2_,iWiXi mod g, diWiy^O, Vj|-u;j| < (logg)'' J 

Notice the resemblance with (7), where multiplication in GF{q) is now replaced by addition 
modulo q (where q can be any integer and does not need to be a prime power). It remains 
unclear to what extent Ajtai's work [Ajt96] relates this problem to the worst-case shortest 
vector problem. It is an open problem whether to believe in the WK assumption. 
Let H : B ^ Zq be a poly-random function. We define 

n{M) = Y^ MbH{b) mod q, (9) 

beB 

=n to be equal to =, and +^1 to be addition modulo q {q plays the role of 2™ in 
MSet-Add-Hash). The proof of the next theorem is similar to the proof of Theorem 2 
in Appendix C. 

Theorem 6. Under the WK assumption, {'H, +-}{■, =1-1) o.^ defined in (9) is multiset colli- 
sion resistant. 

For completeness, we introduce a multiset hash corresponding to parameters n = 2 v"^ 
and I = y/m (see Section 4). Let H : B ^ 7Z^ be a poly-random function. Now, we define 

U{M) = ^ MfeF(6) mod n, 

heB 

='}i to be equal to =, and -{--u to be vector addition modulo n. Theorem 6 holds again if 
we modify the WK assumption by replacing Xj <— Zq by Xi <— ZZ^, Wi G Zq by Wi G ^„, 
and q by n. The main difference is that the Xj's are vectors of length I = y/m. According to 
[GGII96, Sections 2.1 and 2.2]^, if there is a ppt solving the modified WK problem (that 
is it contradicts the modified WK assumption) then, by Ajtai's theorem [Ajt96], there is a 
probabilistic polynomial (in I) algorithm which, for any lattice £ in IR , given an arbitrary 
basis of £, approximates (up to a polynomial factor in /) the length of the shortest vector 
in C. This proves Theorem 3. The worst-case shortest vector problem is believed to be 
hard, see [GGII96] for more discussion. 

E Proof of Improved Offline Checker 

In this appendix, we prove Theorem 4. 

Proof. Suppose the RAM does not behave like valid RAM (i.e. the data value that the 
checker reads from a particular address is not the same data value that the checker most 



^ Notice that the matrix with columns xi is in ^l^''^ and that the vector with entries Wi is unequal to 
zero and has Euclidean norm polynomial in I — \/m. 
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recently wrote to that address). Let W be the multiset of triples written to RAM and let 
R be the multiset of triples read from RAM. We will prove that W ^ R. 

Consider the put and get operations that occur on an address as occurring on a 
timeline. To avoid confusion with the values of Timer, we express this timeline in terms 
of processor cycles. Let xi be the cycle of the first incorrect get operation. Suppose the 
checker reads the pair {vi,ti) from address a at xi. If there does not exist a cycle at which 
writes the pair {vi,ti) to address a, then W ^ R and we are done. 

Suppose there is a cycle X2 when (vi, ti) is first written to address a. Because of line 3 
in the get operation, the values of time stamps of all of the writes to a after xi are 
strictly greater than ti. Because the time stamps at xi and X2 are the same, and since 
put operations and get operations do not occur on the same cycle, X2 occurs before xi 
{x2 < xi). Let X3 be the cycle of the first read from a after X2- Notice that xi is a read 
after X2, so xi> x^. If xi were equal to X3, then the data value most recently written to 
a, i.e. fi, would be read at xi. This contradicts the assumption that xi is an incorrect 
read. Therefore, xi > X3. 

Because the read at cycle xi is the first incorrect read, the read at cycle X3 is a correct 
read. So the read at X3 reads the same pair that was written at X2- Again, because of 
line 3 in the get operation, the values of time stamps of all the writes to a after X3 are 
strictly greater than ti. Therefore, (t'i,ti) cannot be written after X3. Because X2 is the 
first cycle on which (wi,ti) is written to a, (ui,ii) cannot be written before X2- Because 
X3 is the first read from a after X2, and two writes to an address always have a read from 
that address between them, (fi,ti) cannot be written between X2 and X3. Therefore, the 
pair (wi,ti) is written only once, but it is read at xi and X3. Therefore, W ^ R. 
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